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INITIATIVE 


The purpose of this report is to describe 
the development of the ADL Initiative 
Distributed Learning Capability Maturity 
Model (DL-CMM), illustrate its major 
components, and explain how it can be 
used to improve processes. The DL-CMM 
is a tool to appraise an organization’s or 
enterprise’s capabilities in its distributed 
learning processes and functions. This 
document describes each step of the 
model’s development and provides guidance 
for the preparation and use of the model for 
appraisal. 


The CMM has the potential to provide a 
“total view” of the current distributed 
learning status of a _ participating 
organization. It shows the resources, 
expertise and capabilities an organization 
needs to optimize its use of distributed 
learning. Where these elements are 
lacking, the DL-CMM provides perspective 
on what must be implemented to reach 
the next level of capability. 


Organizations, enterprises, and projects 
endeavoring to implement distributed 
learning systems and processes are 
candidates for using the DL-CMM. Key 
personnel in senior leadership, learning 
design, information technology and data 
management and others invested in 
the distributed learning implementation 
process can use the model to gauge 
their current maturity and improve their 
distributed learning processes. 


ADL INITIATIVE 
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INTRODUCTION 


Advanced Distributed Learning (ADL) Initiative efforts around 
the world are trying to determine a) what capabilities their 
respective organizations have achieved; b) what capabilities 
other member organizations have achieved to maximize efficient 
capability sharing; and c) what capabilities need attention in 
their future efforts. One way to achieve this is by creating a 
capability maturity model (CMM) to clarify the general areas 
of focus for distributed learning (DL) and the developmental 
maturity processes needed to achieve optimal DL capabilities. 
This paper traces the development of the DL-CMM to explain the 
pathway to success for full DL maturity. This model will provide 
the roadmap and the justification to our senior leadership 
regarding where the best return on investment (ROI) lies and 
how to achieve it. 


Distributed learning is defined as a “unified, technology- 
enabled interconnected learning paradigm” (Graesser, Hu, & 
Ritter, 2019, p.18). However, the term is used to define many 
different elements of distributed learning practices. It is often 
seen as synonymous with distance education and any teaching 
that is not face-to-face. Some describe it as a shift from 
print-based distance education to the use of information and 
communication technologies for course delivery. Others view 
it in terms of human-computer interaction or as distributed 
cognition, where learning is viewed as distributed among 
individuals. More broadly, distributed learning breaks down 
the traditional boundaries between face-to-face and open and 
distance education. Learning is viewed as a shared enterprise 
distributed between individuals in diverse contexts and not tied 
to formal institutional settings (Lea & Nicoll, 2013). 


The DL-CMM provides a systematic method for understanding 
existing distributed learning maturity by considering the multiple 
areas covered by these broad definitions. This report includes a 
review of other CMMs that address learning and technical processes 
critical to the implementation of distributed learning within an 
organization. Grounding the DL-CMM on previous models will 
provide a useful self-analysis tool as well as a significant addition 
to the distributed learning knowledge base. 


A CMM is a multi-dimensional development model used for 
measuring the degree of formality and optimization (maturity) of 
an organization's processes. It has several categories that together 
make up the major tasks or focus areas within the model. The 
model is typically used to help assess or benchmark the level of 
maturity of expected practices in an organization. 


Main Goals for Maturity Models 


To measure the maturity of the process 
under consideration (i.e. assign a level to 
organization's existing process) 


To provide a mechanism of learning to 
improve the maturity level (Randeree, 
Mahal, & Narwani, 2012) 


To compare the maturity of an 
organization to other organizations and/ 
or best practices 
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Generally, a CMM includes main components 
frequently called dimensions. Under each 
category there are several subcategories 
that drill down the activities within each 
key area. Each subcategory then has five 
levels of maturity (see Figure 1). 


These levels have their roots in the 
quality management field, beginning 
with the Quality Management Maturity 
Grid (QMMG), which describes the typical 
behavior exhibited by an organization at 
each of the five levels of maturity. The 
maturity grid suggests that companies are 
likely to evolve through the five phases 
of quality management success (Fraser, 
Moultrie, & Gregory, 2002). Breaking 
down ADL Initiative’s maturity levels using 
this method can help to create a roadmap 
toward process improvement. 
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Basic Structure of a CMM 


“Dimension 


Key Process 
Area (KPA) 


Level 0: 
Initial 


BENEFITS 


A SINGLE UNIFIED MODEL that integrates 
the best guidance available from recognized 
standards and practices related to distributed 


learning processes. 


ORTHOGONAL - The model is integrated and 
lateVanatealy4cle Mm -x-(olpMcolUlcer-mexelaldalelUic-tom lun) ole)at-lale 
perspectives. Stakeholders can choose those 


areas relevant to their needs. 


03 


ITERATIVELY DEVELOPED by building on the 
previous and current versions to improve the 


model. 


Level 1: 
Managed 


Level 2: 
Defined 


Level 3: 
Quantitative 


Level 4: 
Optimized 


Figure 1: CMM Levels and Key Process Areas 
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LIMITATIONS 


OVERSIMPLIFIES REALITY - Models are 
characterized as step-by-step recipes that often 


lack empirical evidence. 


IGNORES ALTERNATIVE PATHS TO MATURITY 
that might be just as effective (Teo & King, 1997). 


03 


STANDARDIZED CMM BLUEPRINTS used by 
multiple models may not reflect the processes 


specific to the needs of specific organizations. 
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MODEL DEVELOPMENT 


To mitigate criticisms of the model and establish a realistic set 
of requirements for the design of the model, development is 
increasingly done from a design process and perspective (é.g., 
Becker, 2009; Proenca, 2016; Rédlinger, POppelbuB, & Becker, 
2012). 


The DL-CMM model follows the seven design science guidelines 
originally established by Hevner et al. (2004). Design science is 
meant to develop a problem-solving artifact that will contribute 
to current research. 


Step Two 


Step One 


Q 


Problem 
et=Yaraiaerlacey ay 
and purpose for 
ivatemanveye (=) 


a 
&: 


Problem 
Requirements 
and comparison 
with current 
models 


Step Three 


This method was chosen to develop a CMM as an artifact to 
improve an organization’s problem-solving capability. The seven 
guidelines for design science were revised to meet the specific 
needs of creating a CMM (from Becker, 2009). 


Figure 2 depicts the steps used to develop the DL-CMM. This 
technical report describes the development process culminating 
with Step 5. Future steps will include a pilot study and the 
published results of that study. 


Step Four 


</> 


Iterative design and 
development, 
including 
dimensions, key 
process areas and 


Demonstrate 
and evaluate 
new ADL 
Maturity Model 


levels 


Step Seven 


Document, 
design, and 
publish results 


Step Six 
~A 
Evaluate further 
through pilot of 
maturity assessment 


tool in a specific 
domain 


Step Five 


Develop report 
of model 
development 
process 


Figure 2: DL-CMM Development Steps 
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Purpose and Problem Identification 


The main questions driving the development of this model are: 


How is “maturity” defined for distributed 


learning organizations, given the recent 
revolutions in learning science and 
technology? 


What is the current level of maturity fora 


given Defense, Government, or Coalition 
distributed learning organization? 


What are the relative strengths and 
weaknesses of various Defense, 
Government, or Coalition distributed 
learning organizations? 


What steps can an organization take to 
improve its maturity level? 


What are the challenges to implementing 
this model? 


Review of Existing Models 


Support for distributed learning processes is becoming 
indispensable for many organizations. Responsibility for the 
effective and efficient design and use of distributed learning 
processes lies with the organization’s management. The main 
goal is to continually improve the processes necessary for an 
efficient distributed learning organization. 


To identify the requirements for developing a CMM, an 
extensive review and comparison of other existing maturity 
models was undertaken to determine common components and 
processes (see Appendix B for a list of reviewed models). Three 
evaluation criteria were used to select the existing models: 1) 
Documentation includes reference to a developed model; 2) 
Documentation indicates steps of the design process; and 3) 
Validity testing passes at least the weak test as described by Aho 


(2009). The weak test is passed when an organization’s manager 
is ready to use the CMM for decision-making. The strong test is 
passed when the CMM is shown to have improved performance in 
the organization. Table 1 lists the models that met these criteria 
and were used for comparison. 


Although there is a wealth of literature about CMMs, many 
did not meet at least one of the three criteria. Mature models 
were likelier to meet these criteria if they followed an iterative 
process of application and development with publication of one 
or more documents describing the process and/or guidelines for 
implementation. Some models were also excluded due to lack 
of public access to documents, such as Gartner’s Five Stage 
Maturity Model for Logistics Excellence (Lisica & Gonzalez, 
2019). 
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MODELS 


CAPABILITY MATURITY 
MODEL/CAPABILITY 
MATURITY MODEL 
INTEGRATED (CMM, 
CMMI) 


E-LEARNING MATURITY 
MODEL (eMM) 


THE DATA WAREHOUSE 
INSTITUTE’S (TDWI's) 
ANALYTICS MATURITY 
MODEL 


PEOPLE CAPABILITY 
MATURITY MODEL 


ENTERPRISE SPICE 


DEFENCE TECHNOLOGY 
ENABLED LEARNING 
(TEL) MATURITY 
MODEL 


ENTERPRISE 
ARCHITECTURE 
MANAGEMENT 
MATURITY 
FRAMEWORK (EAMMF) 


STATED 
PURPOSE 


Develop a tool for 
evaluation of software 
organizations 


Support higher education 
e-learning systems 


Focus on business 
intelligence, data 
warehousing, and more 
recently the emergence 
of analytics and machine 
learning 


Framework to help 
organizations improve 
workplace competencies 


Domain-independent 
process assessment 
model for enterprise- 
wide assessments and 
improvements 


To assess TEL maturity on 
training units, to identify 
shortfalls that prevent 
optimization of training 


Provides a benchmark 
that can help federal 
departments and agencies 
plan for and measure EA 
program maturity 


COMPARISON WITH 
EXISTING MODELS 


First CMM model based 
on Crosby's Quality 
Maturity Grid for software 
development 


Concepts from CMM 

and Software Process 
Improvement and 
Capability Determination 
(SPICE) 


The original BI model 


(Eckerson, 2009) was based 


on CMM, later adapted to 
current Analytics model 
(Halper & Stodder, 2014) 


Complements the CMMI 
model and expands on 
Watts Humphrey's (1997) 
people maturity model 


Built on CMMI versions, 
ISO/IEC 15504-2, and 
CMM v. 2, developed by 
the US Federal Aviation 
Administration 


Gartner Digital Maturity 
Research, Deloitte Digital 
Maturity Transformation, 
UK Cabinet Office 

GDS (The 7 Lenses of 
Transformation) 


CMMI for Development v. 
1.2; OMB EA Assessment 
Framework, version 3.1; 
GAO ITIM Framework, v. 
1.1 


TABLE 1: Comparison of Reviewed Capability Maturity Models 


ITERATIVE 
ID) =SILE]N| 


Literature reviews 
method Expert 
interviews 


Case studies of applications 


and workshops to modify 
the model 


First iteration (Eckerson, 
2003) was technology- 
focused; later versions 


added cultural and business 


aspects (Eckerson, 2009; 
Halper & Stodder, 2014) 


Version 1 implemented 

for six years, followed by 
Version 2, which was based 
on continuing feedback and 


experience from use around 


the world. 


The SPICE User Group 


released draft models in 


2008 and 2009 for feedback 


Not described in existing 
documents 


Versions 1,1.1 were used to 


perform government-wide 


reviews The current version 


2 builds on the previous 


versions and feedback from 


stakeholders 


Delphi 


EVALUATION 
PROCESS 


Preliminary versions 
accessible for review, 
workshop with 200 
experts, widespread 
application 


First version validated at 
New Zealand university 
workshops. Applications in 
several organizations 


Provides a tool for quick 


overview of an organization’s 
BI ecosystem. Validity is not 


addressed 


Implemented by many 
different organizations 
(e.g., Boeing, Ericsson, 
Lockheed Martin). Has 
been translated into many 
different languages 


The initial model was 
published in 2010. 
Currently in the second 
phase of development: 
deployment and usage; 
phase 3 is still in progress 


Will be integrated into 
the MMAT to capture TEL 
capabilities on a unit 


Review of Enterprise 
Architecture CMMs gives 
EAMMF the highest rating 
among four well known 
models (Suchaiya & 
Keretho, 2018) 


0g 


a0) 5] bs) =D) 
RESULTS 


573-page report and 246- 
page report detailing the 
analysis procedure 


Published descriptions 

of the model and design 
procedure available to the 
public; examples 


Published descriptions of 
the model, instructions 
for evaluating 
benchmark scores for 

35 questions within five 
dimensions 


Version 1 (1995); 
Version 2 (2005). 
Version 2 explains model 
development, description 
of the framework and 
components, as well as 
description of how to 
implement the model 


A technical report was 
published in 2010, no 
further publications were 
found on the Enterprise 
SPICE website 


Still a working exemplar. 
No published results. 


Three published reviews 
of federal departments 
and agencies done using 
v.1.1. 
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Capability Maturity Model (CMM, CMMI) 


Most current models are based on the CMM developed by Carnegie Mellon 
in the late 1980s (Paulk et al., 1995) and continued with the Capability 
Maturity Model Integration (CMMI). Both versions of the CMMs specifically 
focused on improving organizational processes (CMMI Product Team, 2006). 
Many organizations in diverse sectors have adopted and revised the original 
CMM to improve process capability maturity. The original CMM describes an 
improvement path from ad hoc, unmanaged processes to disciplined, mature 
processes that improve organizational quality and effectiveness. Figure 3 lists 
the five levels of organizational maturity and the characteristic used in most 
existing models. Each maturity level contains key process areas 

(KPAs) grouped into common features (categories), which 05 

0 Optimizing 


in turn contain key practices. Goals for each 
KPA are reached by implementing 
and/or institutionalizing the key 
Process change 
0) 3 eh Managed management 
* Technology change 
* Quality management 
mi 4 - D€fined 
sl 


practices within 
management 
* Quantitative process 


each category. 
+ Defect prevention 
0 1 management 
ie Repeatable 


iia Initial/Ad hoc 


+ Peerreviews 

+ Intergroup coordination 

+ Product engineering 

+ Integrated software 
engineering 

* Training program 

+ Organization process definition 

* Organization process focus 


+ Configuration management 
* Quality assurance 

+ Subcontractmanagement 
+ Project tracking/oversight 
+ Project planning 


* No Key Process : 
+ Requirements management 


Areas 


People Capability Maturity Model (P-CMM) 


The P-CMM (Curtis et al., 1995, 2009) focuses on continuously improving the 
management and development of human resources within an organization. It 
is widely used by various types of organizations (e.g., information technology, 
pharmaceuticals, defense agencies, etc.) and has been translated into Japanese, 
Chinese and other languages. 


Based on Watts Humphrey’s (1989) evolutionary approach to process maturity 
as well as the original CMM framework (Paulk et al., 1993, the P-CMM’s five 
progressive levels are meant to transform an organization’s culture by providing 


10) 


a pathway for organizations to implement practices for 
attracting, developing, motivating, and retaining its 
human capital. Like the Humphreys model the P-CMM 
integrates three domains: processes, total quality 
management practices, and organizational change. Its 
targeted domain is workforce management processes. 


Using the CMM architecture the P-CMM’s top layer 
consists of five maturity levels, each representing a 
well-defined evolutionary plateau institutionalizing 
that level of capability for improving the organization's 
workforce. Each maturity level is composed of several 
process areas which contain goals that establish that 
process area’s capability when the goals are met. 
Process areas are defined as a set of related practices 
that satisfy the goals when they are practiced together. 
The P-CMM has 22 process areas that are linked to 
specific levels between 2 and 5. The first level does not 
contain any process areas. 


The Data Warehouse Institute's 
(TDWI) Analytics Maturity Model 


TDWI's Analytics Maturity Model (Halper & Stodder, 
2014) is an offshoot of its Business Intelligence 
Maturity Model (BIMM, Eckerson, 2009), developed to 
help organizations evolve their analytic strategies as 
an essential element of their business decisions. The 
original model was focused on the technical aspects 
of an organization’s BI maturity, whereas this model 
also incorporates maturity for an organization’s culture, 
leadership, governance, and data management beyond 
the technological infrastructure. There are five maturity 
levels: nascent, pre-adoption, early adoption, corporate 
adoption, and mature/visionary. Within this maturity 
continuum the model describes a chasm between the 
early adoption and corporate adoption levels. 
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Enterprises must address four challenges in order to cross this 
chasm successfully: adequate funding, data management and 
governance, skill sets, and cultural and political issues. Maturity 
is measured on five dimensions: organization, data management, 
infrastructure, analytics, and governance. Evaluation of the 
organization’s analytics maturity is done by administering a 
35-question survey across these dimensions that is scored per 
dimension and stage to allow an organization to gauge in which 
dimensions they are more or less mature. This model provides a 
quick way to assess maturity. It does not gauge precisely where an 
organization is positioned in the continuum due to the limited set 
of survey questions covering five broad categories. 


Enterprise SPICE 


Enterprise Software Process Improvement and Capability 
Determination (Enterprise SPICE Project Team, 2010) integrates 
codified standards and the ISO/IEC 15504 assessment framework. 
ISO/IEC TR 15504-7:2008 specifies the conditions for, and contains 
guidance on constructing an Organizational Maturity Model, 
performing assessments of organizational maturity, and applying 
the ratings for process improvement and capability determination. 


Benefits of Enterprise SPICE 


» One unified and comprehensive model - no need for 
separate models for each dimension/component of 
the organization 


» Pick and choose areas relevant to the organization 


» Authoritativ 
and mapped 


widely recognized standards are used 


» Reduced costs — training, improvement, assessment is 
only needed for one model 


» Certification —the model will provide certification from 
accredited bodies 


as 


.Level n 


Process attributes are 
defined at each level 


Process 1, Process 2, ..Process n 


PROCESS DIMENSION 
Figure 4: Structure of SPICE Conformant Models 


CAPABILITY SCALE 


Nv 
zo) 
> 
a 
— 
> 
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> 
o 
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SPICE is a domain independent model that brings together the 
many improvement models, standards, and approaches that target 
specific parts of organizations into one integrated model. 


SPICE conformant models have two axes to denote processes 
and process capabilities. The process axis contains the processes 
grouped into dimensions. The process capabilities axis allows 
capabilities of each process to be measured independently (see 
Figure 4). The model is still in phase two of three phases: initial 
release, deployment and usage, and subsequent releases. There 
have been no new publications released since 2010 but the model 
provides a SPICE-based improvement framework as a foundation 
for developing other models (e.g., Marshall’s E-Learning Model). 
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E-Learning Maturity Model (eMM) 


The eMM is a framework designed to support educational institutions 
to improve their organizational and technological capabilities for 
complex and changing learning environments (Marshall, 2010). 
The main goal of the framework is to help organizations collaborate 
through joint benchmarking projects to identify common areas 
needing improvement and to share examples from their own 
organizations’ improvement efforts. The model has been applied 
to several Australian universities (eight as of 2013) with the goal 
of further applications that will build a corpus of assessments that 
identify good practices as well as challenges (Marshall, 2013). 


The eMM framework is modeled on concepts from the CMM 
(Paulk et al., 1993) and the Enterprise SPICE model (Enterprise 
SPICE Project Team, 2002). In the eMM, the Capability concept is 
defined as the organization’s ability to design, develop, and deploy 
e-learning to meet the needs of the students, staff and institution. 
From SPICE, the eMM divides Capabilities into five major Process 
Areas: Learning, Development, Support, Evaluation, Organization. 
Each Process Area is further divided into Practices that provide 
the benchmarks for assessment. These Practices are derived from 
widely accepted guidelines found in Chickering and Gamson’s 
(1987) “Seven Principles” and the Institute for Higher Education 
Policy’s “Quality on the Line” benchmarks (Phipps & Merisotis, 
2000). Rather than the maturity levels found in the CMM (see Table 
1), the eMM uses Dimensions to describe five capability types for 
each of the Practices under the Process Areas: Delivery, Planning, 
Definition, Management, and Optimization. Finally, each of the 
five Dimensions for the Practices is assessed using a color-coded 
scheme standing for Fully Adequate, Largely Adequate, Partially 
Adequate, Not Adequate, and Not Assessed. 
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Defence Technology Enabled Learning 
(TEL) Maturity Model 


The Defence TEL Maturity Model is being developed to provide 
a method of evaluation to examine unit training management 
and governance functions, strategies, and processes that enable 
or hinder technology enabled learning (TEL). A maturity model 
was chosen to capture current TEL capabilities that affect method 
and media selection, allow commanders to self-assess their levels 
of TEL maturity across several predefined functions, and reveal 
evidence of areas for continuous improvement and TEL investment 
for planning and budgeting purposes. The current model has four 
maturity levels:ad hoc/initial, repeatable, defined, and managed 
and measured. Six Functional Areas and associated Subfunctional 
areas are addressed (see Table 2). 


TABLE 2: DTEC MM Functional Areas and Subfunctional Areas 


lO laveud (e)arel| : 
aie | STU] obs at U areii(o)atel-\aetelsy 
Areas 


GOVERNANCE None 

STRATEGY Data Strategy, Digital Literacy Strategy; Investment Strategy; Innovation 
Strategy 

USER Stakeholder Map; Digital Literacy Levels (based on DSTL Digital Mapping 
Tool); Accessibiity of Materials; Design, Delivery, and Management function 

LEADERSHIP Vision/Mission Statement; Digital Awareness; Promotion of TEL and 
Innovation; Activities Orientated Across Elements 

INTEGRATION- Architecture; Appropriate Range of Devices; Levels of Connectivity; 

TECHNOLOGY Standards Recognition and Implementation; Collaboration with Other Units 
and Entities 

INTEGRATION- Course Methods and Media Analysis; Assessment; Range of Medium and 

CONTENT Supporting Elements; Analytics; Collaboration with Other Units and Entities 


ADL INITIATIVE 


In comparison to the DL-CMM, the Defence TEL Maturity Model can 
be modified to provide the ability to assess the DL capabilities of 
training units to provide courses within the larger DL infrastructure. 
Figure 5 provides an overview of the differences and similarities of 
the two models. Although different in scope both models could be 
used in tandem to provide complementary evaluations to benefit 
both enterprise and course level planning and budgeting. 


DEFENCE TECHNOLOGY ENABLED LEARNING 
MATURITY MODEL (DTEL-MM) 


DISTRIBUTED LEARNING 
CAPABILITY MATURITY MODEL (DL-CMM) 


SCOPE Enterprise Level Course Level 


Provide a “total view” of current distributed 
learning capabilities to facilitate self-appraisal 
and provide discussion points 


Self-assess TEL maturity levels across 
functions and benchmark across units 
within their sector or service 


PURPOSE 


Capability Maturity Model Integrated; 
Enterprise SPICE; People Capability Maturity 
Model 


KEY SOURCE 
MODELS 


Gartner Digital Maturity Research; Deloitte 
Digital Maturity Research 


a DIMENSIONS & er . : 7 
=:] KEY PROCESS Distributed learning processes and The 7 Lenses of Transformation; Academic 
aed AREAS technologies; Academic sources sources 
Senior leaders, learning designers, 
Ba STAKEHOLDERS schoolhouse management, analysis ClO Commanders 


office, Unit level leadership 


Figure 5: Comparison of DL-CMM to DTEL-MM 


Enterprise Architecture Management 
Maturity Framework (EAMMF) v.1.1, v2 


The Enterprise Architecture Management Maturity Framework 
(EAMMF) version 1.1 was developed and published in 2003 by 
the Government Accounting Office (GAO) as a way to measure 
Enterprise Architecture (EA) program maturity in Federal agencies. 
It was based on guidance from the 1996 Clinger-Cohen Act and 
subsequent Federal CIO Council’s Federal Enterprise Architecture 
Framework (FEAF) published in 1998. EAMMF version 2 came out 
in 2010. The GAO (2010) describes EA as an “essential tool for 
effectively and efficiently engineering business or mission processes 
and for implementing and evolving supporting systems” (p. 1). 


Federal EA was developed according to a collection of five reference 
models that were established by OMB in 2002: 
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» Business Reference Model (BRM): Lines of business, 


agencies, customers, and partners 


» Performance Reference Model (PRM): Inputs, outputs, and 
outcomes, uniquely tailored performance indicators 


» Data Reference Model (DRM): Business focused data 
standardization, cross agency information exchanges 


» Service Component Reference Model (SRM): Service 
domains, service types, business and service components 


» Technical Reference Model (TRM): Service component 
interfaces, interoperability, technologies, recommendations 


These models were designed to provide a way to improve 
cross-agency analysis and identify gaps, duplication of efforts, 
opportunities for collaboration, and interoperability within and 
across agencies and departments (GAO, 2010; OMB, 2009; OMB 
2012). 


Figure 6 provides an overview of EAMMF version 2 with a description 
of each of 7 stages of maturity. Stage 0 has no core elements 
because EA awareness is emerging. There are four sets of critical 
success attributes (called representations) and 59 core elements. 
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fae?) epee erase 1 ‘ 
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1 . 
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Figure 6: EAMMF Overview with 7 Stages of Maturity (GAO, 2010) 
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TABLE 3: EAMMF Representations and Critical Success Attributes 


Critical Success Attributes 


Demonstrates commitment; Provides capability to meet 
commitment; Demonstrates satisfaction of commitment; 
Verifies satisfaction of commitment 


Representations 


EA MANAGEMENT ACTION 
REPRESENTATION 


EA FUNCTIONAL AREA 
REPRESENTATION 


Governance; Content; Use; Measurement 


OMB CAPABILITY AREA 
REPRESENTATION 


Completion; Use; Results 


EA ENABLER 
REPRESENTATION 


Leadership; People, Processes; Tools 


Table 3 names the four representation categories of critical success 
attributes and their associated attributes. 


Version 1.1 was used to perform two major reviews to evaluate 
the status of EA in the federal government (Gaver, 2010). The 
first review was federal-wide and was based on four surveys done 
between 2001-2006. If assumed to be a valid measure of EA, the 
results showed that federal EA did not mature as it should. Between 
2001 -2010, GAO also measured agency-specific maturity and found 
many problems that needed to be addressed within each agency 
(Gaver, 2010). 


Version 2 was a major update of the previous two versions. GAO 
used feedback from the stakeholders they surveyed during their 
evaluations to improve the model and expand its scope. A recent 
review gave the EAMMF a rating of 100% compared to other current 
EA maturity frameworks as analyzed by a set of critical success 
factors: scoping and purpose, communication and common language, 
business driven approach, commitment, development methodology 
and tools supported, EA models and artifacts, EA governance, project 
program management, assessment and evaluation, IT investment and 
acquisition, skilled team, training, and education, and organizational 
culture (Suchaiya & Keretho, 2018). 
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lterative Development 


The DL-CMM’s framework was developed using the maturity 
grids found by Maier, Moultrie, & Clarkson (2010). The authors 
reviewed 24 existing maturity grids used for managing and 
improving organizational capabilities. This framework was chosen 
for two reasons. First, unlike models such as the CMM, it does 
not need to be evolutionary and can cover best practices for the 
entire organization rather than specific processes like software 
development and acquisition. Second, the model is structured 
like a matrix grid, with each cell describing the characteristic trait 
necessary for performance at each level. Table 2 describes the key 
elements derived from the review of existing models to develop 
the DL-CMM. 


The DL-CMM based its maturity levels on the original CMM’s (Paulk, 
1995) five maturity levels to represent the plateaus reached for 
each KPA rather than the six capability levels for continuous 
representation described in the new CMMI version. The reason 
for this is that the DL-CMM is an enterprise-level model that is 
built to provide a variety of organizations the ability to look at the 
generic distributed learning practices required for maturity in their 
organization. 


All six reviewed models grouped the key process areas they were 
measuring into several components or dimensions. However, CMM, 
CMMI, and other software development maturity models additionally 
grouped the process areas and dimensions into each maturity 
level. Enterprise SPICE and eMM did not take this evolutionary 
approach. Process areas were grouped into dimensions or 
categories but were independent from maturity levels. The DL-CMM 
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from maturity levels. The DL-CMM features five dimensions 
- Commitment, Human Infrastructure, Data Infrastructure, 
Technological Infrastructure, and Design — that were derived from 
the ADL Initiative’s history of research and development in this 
area. In particular, the book Modernizing Learning (Vogel-Walcutt 
& Schatz, 2019) provided a compendium of multidisciplinary works 
on distributed learning that were mined for key dimensions and 
practice areas. 


Another key element of model development involved detailed 
interviews and discussions with a five-member team of Subject 
Matter Experts (SMEs). Over a five-month review cycle, SMEs 
and the internal research team reviewed and discussed each key- 
dimension area and elicited key process areas as they related to 
each dimension for clarity, conciseness, and completeness. 


TABLE 3: DL-CMM Elements Derived from the Review 


Capability This is what the model measures to analyze and improve processes. The 
DL-CMM builds on the more general organizational and technological 
aspects of CMMs to include distributed learning processes for design and 
delivery of learning to meet the needs of students, staff and the learning 
organization. (Marshall, 2010) 

Dimension Dimensions are specific capability or process areas that structure key 
categories of distributed learning. They should be exhaustive and distinct, 
with each dimension further specified by several key process areas 


Key Process Enterprise SPICE process areas fall under process dimensions. CMM and 

Areas CMMI process areas are a cluster of related practices in an area that satisfy 
a set of goals when they are implemented collectively 

Assessment The assessment approach can be qualitative (using descriptions) or 

quantitative using tools such as Likert scales 
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Demonstrate and 
Evaluate 


The DL-CMM was demonstrated to key stakeholders at 
Innovation, Instruction, Implementation Fest (iFEST) event in 
2019. This annual conference is devoted to sharing the latest 
distributed learning innovations. It was also reviewed by 
stakeholders at the 2019 meeting of the NATO Training Group 
Task Group on Individual Training and Education Developments 
(NTGTG IT&ED). The model was presented as a tabletop rubric 
that could be shared and discussed among the evaluators. They 
found the rubric format easier to share than other formats 
(e.g., Likert surveys). The DL-CMM will be sent to all member 
nations to garner feedback at their next meeting. 


Report 


This report documents the history of the DL-CMM’s development 
and use of in-house reviews, as well as a small focus group 
evaluation to gauge the reliability and validity of the model. 
Further testing and evaluation will be conducted to accurately 
measure the extent to which the model is a reliable indicator 
of an organization’s maturity at each of the five levels for each 
KPA. 
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Structure of the DL-CMM 


The core structure of the DL-CMM is composed of three major 
sections; 1.) Dimensions, 2.) KPAs, and 3.) Levels (See Figure 5). 
Dimensions represent the breadth of core organizational structures 
upon which practices, structures, responsibilities, or approaches 
can be measured. 


Dimensions of the DL-CMM 
COMMITMENT: Collective coordination across 
communities 


DESIGN: Design of learning components, systems, 
and environments built on learning science and 
learning engineering 


HUMAN INFRASTRUCTURE: Multidisciplinary 
coordination of human contributors to the 
learning ecosystem 


TECHNOLOGICAL INFRASTRUCTURE: 
Technology-enabled learning architecture: 
instructional systems, interoperability standards, 
software services 


DATA INFRASTRUCTURE: Digital infrastructure to 
promote data sharing and usage 
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Architecture of the model 


The DL-CMM contains five dimensions that are orthogonal to each 
other. Orthogonality is important to avoid conflicting outcomes from 
data gathering. By creating severable outcomes, the model can be 
tailored to the resources of an organization and its data can feed 
assessments that consider actions taken to determine effectiveness. 


Cost to implement the model 


The model requires the participation of senior leadership, personnel 
proficient in analysis, and access to material and subject matter 
expertise across a range of disciplines in a given organization. This 
can be accomplished with minimal investment. 


Access to SMEs is a known cost for any analysis, so organizations 
should carefully plan SME participation to lessen significant impact 
of organizational workflow. 


Value of the model 


When properly applied, the DL-CMM will provide both vertical data 
points on how leadership vision is being implemented and horizontal 
data points on how different activities are integrated to provide 
mature learning ecosystems. 


Leadership should be particularly interested in the outcomes of how 
data is being leveraged in relation to leadership intent. The KPAs for 
data provide insight on how knowledge can be captured and inform 
processes. Without strong infrastructure and management, that 
knowledge will not promulgate into positive impacts on individual 
efforts. 


These evaluative criteria should be interpreted individually based on 
organizational goals. The intention of the model is to communicate 
important organizational components at various levels of maturity 
that can be applied and validated across a wide swath of domains 
and organizational levels. 
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Structure of the DL-CMM 


Dimension 
Key Process Level 0: 
Area (KPA) Initial 
KPA 1 


COMMITMENT 


DESIGN 


HUMAN 
INFRASTRUCTURE e 


TECHNOLOGICAL 
INFRASTRUCTURE 


DATA , 
INFRASTRUCTURE e 


KEY PROCESS AREAS 


Vision 
Professional 
Development 


Learning Policy 
Learning Environment 


Pedagogy, Andragogy, 
Heutagogy 


Workforce Development 


Professional 
Development Programs 


Mentoring & Coaching 


Identification of Training 
Requirements 


Governance 
Extensibility 


Ubiquitous Learning 
Environments 


Training Management 


Data Strategy 

Data Analysis 

Data Management 
Data-Driven Decisions 


Level 1: 
Managed 


Financial Planning 


| 

| 
Granularity of Training | 
Records 


Business Rules 
Instructional Design 
Curriculum Design 


Assessment Design 


Knowledge 
Management 


Learning Resources 
Quality Assurance 
Change Management 
Standards 


Security 
Privacy 


Learning Identity 
Management 


Data Interoperability 
Human Resources 
Data Rights 


Level 2: 
Defined 


Level 4: 
Optimized 


Level 3: 
Quantitative 


LEVEL o: INITIAL 


e Organizational capacity is ad hoc and occasionally 
chaotic 


e Few, if any, defined processes exist 

e Success depends on the abilities, efforts and 
organization of individuals 

LEVEL 1: MANAGED 

e Systems for managing training & education are in place 


e Some areas of distributed learning management are 
applied in isolated projects 


LEVEL 2: DEFINED 
e Sharing of expertise between areas 


e Processes used are codified and there is shared 
responsibility for maintaining these processes 


e Common training programs are implemented among 
areas 


LEVEL 3: QUANTITATIVE 


e Systems and procedures (e.g., learning support, 
instructional design processes) are organization-wide 


e Common datastore for collecting and measuring 
training and education activities 


e Well-defined and consistent metrics to aid training and 
education goals 
LEVEL 4: OPTIMIZED 


e Continuous improvements are adopted by reviewing 
and updating processes through incremental 
improvements of methods and technologies 


Figure 7: Dimensions and Key Process Areas of the DL-CMM 
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HOW TO USE THE MODEL 


When incorporating the DL-CMM, it is critical to tie in senior management 
and business leaders, specifically in communicating the value of 
organizational improvement via the associated risks and benefits. When 
considering these factors keep in mind specific instances of where the 
organization has already failed and illustrate personalized scenarios that 
demonstrate the benefits of scaling maturity. The bottom line is that 
CMM implementation is a large-scale effort and needs buy in from senior 
management and business leaders in order to truly incorporate the change 
needed to reap the potential benefits. Communicating the risks of non- 
incorporation and potential benefits to those large-scale decision makers 
will help educate those equipped to create change. 


We recommend following the structure of the DL-CMM when implementing 
it in your organization. If the model is applied piecemeal or adapted to an 
organization based on the availability of SMEs and leadership, it will still 
offer value but the outcomes may be difficult to map to the referenced 
literature reviews. 


Senior 
_ Leadership — 


Design 


IT 


sri Analysis 
ClO Office 


CIO Office 


e 
a (-Ye1 alate) (e)=4[er-]| Data 
Infrastructure Infrastructure 


Staff and 
‘Unit-Level 
Leadership — 


Schoolhouse Human 
_Management Infrastructure 


Figure 8: Key Stakeholders for Each DL-CMM Dimension 


STEP 1: PREPARE 


» Establish relationship with senior leadership and obtain 
consent on executing study 

» Identify risks and benefits to the organization and present to 
senior management and business leaders 

» Identify stakeholder team to conduct CMM analysis (see 
Figure 6 for recommended stakeholders) 

» Develop milestones and schedule aligned to costs and 
availability of stakeholders 


STEP 2: FIT DL-CMM TO THE 
ORGANIZATION 


» Determine organizational goals 

» Tailor the KPA selection for the study to fit the approved 
schedule and to make the best use of approved 
organizational resources 

» Assess desired organizational level per KPA, as it relates to 
overall organizational goal 


STEP 3: ASSESS MATURITY 


» Ask allocated stakeholders to assess current organizational 
maturity aligned to KPA levels 


STEP 4: IMPLEMENT RESULTS 


» Report findings to senior management and business 
leaders 

» Develop strategies for achieving desired maturity 

» Implement strategies to achieve desired outcomes 

» Document outcomes 


ADL INITIATIVE 


19 


CONCLUSIONS 


Recommendations 


Research conducted by Meta Group (2004) revealed two 
organizational challenges to any initiative for deploying CMMs. 
The first challenge is the perception that distributed learning 
processes are primarily technology driven. A fundamental 
objective of the DL-CMM is to link strategy to organizational 
processes in which technology might play an important 
but supporting role. The second challenge is organizational 
resistance. Deploying the model can change the existing 
infrastructure of power and control. It is important to develop 
a training program for end users to deal with lack of user 
confidence in the system (Frolick et al., 2006). Senior leadership 
must recognize the commitment required to implement the 
maturity model. Otherwise they will become disillusioned and 
withdraw support. They should recognize that the DL-CMM is 
not a single project but a gradual building of skills, awareness 
and technology that must be implemented in iterative phases 
over time (Newman & Logan, 2008). 


» Make the DL-CMM part of the effort to educate senior 
management, so they understand the phases of the 
enterprise distributed learning organization’s journey. 


» Illustrate the risks of not having the DL-CMM, look for 
examples where the organization has already failed because 
of poorly managed distributed learning processes. 


Future Steps 


This is a working exemplar of a living document that continues 
to be developed in an iterative design process. The DL-CMM has 
many potential uses and types of application. To further improve 
the benefits of the model and validate its use, we recommend 
applying it to organizations and through focus group sessions to 
determine whether its objectives are met or not. 
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APPENDIX A: DL-CMM MODEL 


Key Process Area 


VISION: 


Does the organization 
have a vision for 
education and 
training? 


PROFESSIONAL 
DEVELOPMENT: 


What is the maturity 
of policy to develop 
the best possible 
personnel? 


FINANCIAL 
PLANNING: 

Does the organization 
leverage Financial 
Planning for 
education and 
training across the 
enterprise? 


GRANULARITY OF 
TRAINING RECORDS 


Does the organization 
represent education 
and training records 
with proper specificity 
and granularity? 


N10) 


Each subordinate organization 
has its own set of standards 
that guide organizational 
goals, vision, and purpose. 


The workforce receives 
opportunities like taking 
external courses but this is 
not formally tracked. 


Local activities request and 
manage funding to meet their 
own education and training 
needs. 


No representation of 
credentials is present in the 
talent pipeline. No common 
system is used for managing 
credentials. 


Commitment 


Collective coordination across communities 


Level 1 


Each subordinate organization 
has its own set of standards 
that guide organizational 
goals, vision, and purpose. 
There are some established 
channels of communication 
and reporting to enable 
education and training. 


Although effective workers are 
known within the organization 
and provided with extra 
courses, there is no official 
acknowledgment. 


Funding is optimized across 
organizational departments 
by coordinating learning to 
reduce duplication for funding 
to meet their own education 
and training needs. 


Minimum required credentials 
are represented in the talent 
pipeline. Minimally viable 
systems exist for managing 
workforce credentials. 


Level 2 


Each subordinate organization 
has its own set of standards 
that guide organizational 
goals, vision, and purpose. 
There are established 
channels of communication 
and reporting to enable 
education and training. 


Career growth and leadership 
opportunities are extended 
to workers based on talent 
identification. 


The organization holds 
approval authority over local 
funding expenditure but does 
not have stringent oversight 
on quality. 


The talent pipeline includes 
a transcript of credentials 
that manages recertification 
requirements. The 
organization has established 
processes and systems that 
provide deeper insight into 
workforce credentials. 


=\Vi~1 i} 


The organization is making 
progress in transferring the 
philosophy of collective purpose 
to partner organizations. The 
principles and practice of 
leadership are well developed 
among workers and learners. 


The career development 
process is linked to leadership 
development and workforce 
planning tools. 


The organization holds approval 
authority over local funding and 
defines minimum acceptable 
levels of technical quality. 


The talent pipeline manages 
credentials and training 
requirements aligned with 
career growth and workforce 
planning tools. Credentials 
are defined using Credential 
Transparency Description 
Language (CTDL) to enable 
insight into the competencies 
each credential represents. 
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Level 4 


The organization’s policies enable 
partnerships through specifications 
and standards. There is a growing 
commitment to the success of partner 
organizations and mutual ability to 
improve learning, technology, and data 
analytic goals across the enterprise. 


Using workforce planning tools, 
workers are trained to meet 

future organizational needs across 
different functional areas within the 
organization. 


The organization holds approval 
authority over local funding, and 
defines minimum acceptable levels of 
technical quality that meet 3rd party 
accreditation requirements. 


In addition to Level 3 requirements, 
the talent pipeline tracks learning 
experiences, competencies, and 
credentials. These data are shared 
across the organization's other 
functional areas to improve overall 
workforce efficiency. 


ADL INITIATIVE 


Key Process 


Area 


LEARNING POLICY 
Are there policies 
that govern 

the education 

and training of 
individuals within 
the organization? 


LEARNING 
CULTURE 


How mature are 
the organization’s 
education and 
training practices? 


LEARNING 
ENVIRONMENT 


How pervasive 

are resources for 
learning across the 
organization? 


Are there programs 
in place to afford 
access for learners 
to leverage them? 
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There is no formal learning 
policy. 


Teacher-centered training 
techniques are employed 

to meet mandatory 
requirements. Training is 
based on organizational 
policies rather than from the 
needs of the students. 


Learning resources are 
primarily located in physical 
locations. Access to learning is 
based on industrial classroom- 
based models. Emergent 
training requirements are 
reactionary. 


Design 


Design of learning components, systems, and environments 


built on learning science and learning engineering 
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Policy exists but is limited 
and fragmented across 
organizational components. 


Management recognizes 

the need for learners to 

have more ownership of 

their learning. The learner's 
development and career needs 
are starting to be recognized 
as important to the health of 
the organization. 


Online and physical learning 
resources are available as 
stand-alone capabilities. 
Learning resources are 

not connected to other 
organizational resources. 
Training effectiveness 
evaluations are periodically 
conducted to address the 
overall quality of different 
learning environments. 


Level 2 


Policy exists but is not 
consistently enforced across 
the organization. 


Learners are given tangible 
methods for steering their 
own development and 
careers. The organization is 
starting to create an enabling 
environment that nurtures 
learners and recognizes skills 
and abilities. 


Online, mobile, and physical 
learning resources are 
available as stand-alone 
capabilities. Learning 
resources are connected to 
other organizational resources 
using the Experience API 
(xAPI). Learning resources 
may be blended together 
into a single program of 
instruction. 
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Policy is well-defined, 
enforced, and communicated 
throughout the organization. 
Policies are regularly updated 
and integrated with other 
organizational policies. 


A learner-centric culture 

is employed. Learners 
demonstrate accountability 
for their own learning. 

The organization provides 
opportunities to design their 
own learning paths. Staff use 
credentialing and badging to 
recognize achievement. 


Online and physical learning 
resources are available as 
stand-alone capabilities. 
Learning resources are centrally 
managed and connected to 
other organizational resources. 
Formal programs facilitate 
education and training access 
for a widely distributed 
workforce. Learning resources 
are regularly blended into 
programs of instruction. 
Environments are described 
using LRMI metadata. Learner 
performance within the learning 
environment is tracked using 
xAPI. 


Level 4 
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Policy is well-defined, enforced, and 
communicated organization-wide. 
Policies are regularly updated and 
integrated with other organizational 
policies. Learning policies are tied to 
other education and training tools 
within the organization to manage 


conformance. 


A deep culture of learning exists 


and is encouraged by policies and 
promoted by management. Learning 
opportunities are available across 

the organization and the workforce 

is rewarded for participating through 
promotions or incentive programs. 
Processes are in place for piloting new 
training-related tools or technologies. 


Learning centers of excellence and 
a centralized management function 


provide support across different 


learning environments. Policies exist 
to manage the incorporation of new 
training tools or technologies. Blended 
learning opportunities are pervasive 
through distributed access. Learning 
environments are matched to learner 
needs and mission requirements. 
Learning environments are described 


using LRMI metadata. Learner 
performance within the learning 


environment is tracked using xAPI. 
Emergent training requirements are 
proactively developed and learner 


environments are continuously 


evaluated to measure effectiveness. 


ADL INITIATIVE 


Key Process 


Area 


INSTRUCTIONAL 
DESIGN 


Are there 
prescribed 
processes 

for creating 
instructional 
experiences 
within the 
provided learning 
environments? 


CURRICULUM 
DESIGN 


Are there 
prescribed 
processes for 
identifying 

and aligning 
instructional 
topics from the 
organization’s 
mission? 


ASSESSMENT 
DESIGN 


How pervasive is 
the use of learner 
assessment 

data across all 

the different 
experiences a 
learner encounters? 


Is a learner only 
assessed in formal 
environments? 


Level 0 


Policy prescribes the 
processes and methodologies 
used in the creation of basic 
training programs that meet 
knowledge-based objectives. 
There is minimal use of 
assessments. 


Curriculum design policies 

and processes are ad-hoc. 
There are no policies or 
recommended practices for 
controlling the curriculum 
design process. Few processes 
are defined, and success 
depends on individual effort. 


No policies are in place 
relating to the design and 
reporting of assessment. 
Learning analysis metrics 
remain unidentified and 
uncaptured. 


Design (continued) 
Design of learning components, systems, and environments 
built on learning science and learning engineering 


Level 1 


Policy prescribes the 
processes and methodologies 
used in the creation of 
blended training programs 
that meet organizational 
objectives. There is moderate 
use of scaffolding, knowledge 
checks, assessments, and 
other practice exercises. 


Basic curriculum design 
processes are established 

at the course level. The 

focus is to design clear and 
measurable learning outcomes 
at the course level. 


The use of xAPI enabled 
learning assessments 

is sporadic across the 
organization. Data collected 
is primarily focused on 
completions and satisfying 
higher level course 
requirements. 


Level 2 


Policy prescribes the 
processes and methodologies 
to create blended training 
programs that meet 
organizational objectives. 
There is policy-driven use 

of scaffolding, knowledge 
checks, assessments, and 
other practice exercises. 
Active learning methods 

like case studies, hands-on 
practice, or collaborative 
exercises are minimally used 
across the organization. 


Curriculum design policies 
and processes are formally 
documented, standardized, 
and integrated to support 
the design of activities, 
courses, or instructional 
programs designed to meet 
organizational requirements. 


The use of xAPI enabled 
diagnostic assessments 

is moderate across the 
organization. Collected 
assessment data provide more 
granular details of learner 
proficiencies and deficiencies. 
Assessments inform remedial 
learning opportunities. 


=\Vi-1 i} 


Policy prescribes the 

processes and methodologies 
used to create blended 

training programs to meet 
organizational objectives. 
Learners can test out of 
previously learned course 
components. Policy-driven 

use of scaffolding, knowledge 
checks, assessments, and other 
practice exercises are in place. 
Active learning methods like 
case studies, hands-on practice, 
or collaborative exercises are 
moderately used across the 
organization. Policy requires 
review of instructional design 
artifacts to ensure quality. 


Detailed curriculum design 
policies and processes are 
defined, standardized, and 
adopted organization-wide. 
Policies are in place to align 
learning outcomes with 
organizational goals, and a 
process for identifying key 
performance indicators in 
the operational environment 
exists to provide feedback to 
curriculum designers. 


The use of a wide range of 
xAPI enabled assessments 

is institutionalized. Learner 
proficiency is also assessed 
using operational performance 
built around key performance 
indicators in the work 
environment. Assessments 
inform new learning 
opportunities. 
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Level 4 


Policy enables the development of 
learning environments that support 

a high degree of cognitive fidelity to 
support advanced decision-making 
skills. Learners are able to test out of 
previously learned course components. 
Instructional strategies, design artifacts 
(e.g., job duty task analysis, cognitive 
analysis) and assessments are 
digitally archived to support enterprise 
analytics. Policy mandates the review 
of all instructional design artifacts to 
ensure quality. 


Continuous process improvement 

is enabled by policies requiring 
quantitative feedback from key 
performance indicators in the 
operational environment to be available 
to support the curriculum design 
process. Data driven design principles 
are incorporated into the curriculum 
and external resources are available 
to learners to support higher levels of 
learning. 


The use of a wide range of prescriptive 
assessments continually optimizes 

and tailors assessments to individual 
learners. Assessment data from across 
the organization are aggregated to 
make predictions about the learner’s 
future potential. Learner proficiency 

is also assessed using operational 
performance built around key 
performance indicators in the work 
environment. Assessments inform new 
learning opportunities. 


ADL INITIATIVE 


Key Process 


Area 


WORKFORCE 
DEVELOPMENT 


What is the 
maturity of 
workforce 
development, 
training, and 
mentoring 
opportunities? 


PROFESSIONAL 
DEVELOPMENT 
PROGRAMS 


What is the 

maturity of policies 
to develop the best 
possible personnel? 


MENTORING AND 
COACHING 


How consistent 
are the policies 
for mentoring 
and coaching 
programs? 


IDENTIFICATION 
OF TRAINING 
REQUIREMENTS 


Is there a 
standardized set 
of processes for 
capturing feedback 
from operations 

to education and 
training? 


Human Infrastructure 
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The multidisciplinary coordination of human contributions to the learning ecosystem 


N10) 


There is unstructured on-the- 
job learning, minimal informal 
knowledge exchange, and 
very little teamwork. 


The workforce has 
opportunities, such as 
external courses, to improve 
on-the-job skills. 


There is some mentoring 

and coaching, but it is not 
organizationally driven. There 
are mainly individual efforts. 


Requirements management 
is event driven and training 
requirements are updated 
reactively to negative events 
(e.g., accidents). 
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There are some formal 
training programs. Mentoring, 
sporadic knowledge sharing, 
and teamwork are managed 
ad-hoc in the organization. 


Effective workforce 
development programs 

are established but are 

not consistently used or 
universally promoted across 
the workforce. 


A policy for mentoring and 
coaching is being devised but 
is not yet coherent. There is 
no clear delineation between 
mentoring and coaching and 


roles are not clearly allocated. 


Subject Matter Experts define 
training requirements based 
on industry best practices 
and personal experience. 
Requirements vary across 
organizations. 


Level 2 


Training and mentoring 
structures are promoted 
organization-wide. 
Professional development and 
communities-of-practice are 
encouraged. 


Career growth and leadership 
opportunities are extended to 
the workforce based on talent 
identification. 


A distinction is made between 
mentoring and coaching, with 
roles given to the workforce. 
Learners can mentor and 
coach each other, especially 
in tutoring or peer teaching 
roles, with support from the 
organization. 


Subject Matter Experts 
uniformly define training 
requirements based on 
established set processes 
derived from industry 
accepted best practices. 


Level 3 


Workers are empowered to take 


charge of their own training 
and mentoring. Career growth 
opportunities are promoted 
throughout the organization. 


A career development 
process is linked to leadership 
development and workforce 
planning tools. 


There is a developed policy 
for mentoring and coaching 
that is linked to performance 
management and leadership 
development. 


Policies are in place to 
support continuous process 
improvements for revising 
training based on operational 
feedback. Feedback is 
incorporated. Content is 
updated proactively. 


Level 4 


Professional development programs 
are actively promoted and supporting 
systems are in place for identifying 
and aligning professional development 
opportunities across the workforce. 


Using workforce planning tools, 
workers are trained to meet 

future organizational needs across 
different functional areas within the 
organization. 


Systematic formal mentorship 
programs are in place at the enterprise 
level to offer mentoring and coaching 
opportunities. They track and measure 
Participation and training effectiveness 
across the organization. 


Policies are in place to support data- 
driven training requirements that are 
regularly revised. 


ADL INITIATIVE 


Key Process 


Area 


LEARNING 
RESOURCES 


Are there 
organizational 
policies to 
standardize the 
acquisition of 
information? 


QUALITY 
ASSURANCE 


How mature 
are policies and 
processes for 
verification, 
validation, and 
accreditation? 


CHANGE 
MANAGEMENT 


What is the 
maturity of 
organizational 
change 
management 
policies in response 
to mission needs? 


STANDARDS 


Are standards, 
such as ISO, ABED 
Accreditation, 

and Six Sigma, 
followed? 


No policies are in place 
related to learning resource 
requirements. 


Local activities have the 
authority to verify, validate, 
and accredit instructional 
materials and processes. 


No formal change 
management tools, processes, 
or documented initiatives are 
in place. 


Each activity has its own local 
usage of standards. 


Human Infrastructure (continued) 
The multidisciplinary coordination of human contributions to the learning ecosystem 


Organizational policies exist to 
outline technical and usability 
requirements of any acquired 
learning resources. Policies 
are minimally enforced and 
not consistently promoted 
across the organization. 


Local activities have primary 
authority with some oversight 
from the organization. 


Change management is 
performed on an ad-hoc basis 
across the organization on a 
department by department 
basis. 


Local activities have 
recommended standards from 
the parent organization. 


Level 2 


Organizational policies 

exist that outline technical 
specifications and usability 
requirements of any acquired 
learning resources. Policies 
are moderately enforced but 
not consistently promoted 
across the organization. 


The organization has 
common outside accrediting 
requirements. 


Organizational policies 
formalize a change 
management strategy across 
the enterprise. Policies are 
minimally enforced and not 
consistently promoted. 


Local activities have required 
standards from the parent 
organization. 
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Organizational policies 

exist that outline technical 
specifications and usability 
requirements of any acquired 
learning resources. Policies are 
fully enforced and consistently 
promoted across the 
organization. The organization 
provides adequate resources 
and funding for quantitative 
evaluation of learning 
resources. 


The organization proactively 
engages with common outside 
accrediting requirements 


Organizational policies formalize 
a change management strategy 
across the enterprise. Policies 
are moderately enforced but 
not consistently promoted. 


The organization has identified 
common standards for use 
across the organization. 
Adherence to standards 
requirements is moderately 
enforced. 
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Level 4 


Organizational policies outline 
technical specifications and usability 
requirements of any acquired learning 
resources. Policies are fully enforced 
and consistently promoted across 

the organization. The organization 
provides adequate resources and 
funding for quantitative evaluation 

of learning resources. Policies guide 
the establishment of enterprise data 
dictionaries that are continuously 
updated for the organization. Learning 
resources adhere to the standards 
outlined in the data dictionary. 


The organization has common outside 
accrediting requirements as a cultural 
norm that is incentivized. 


Organizational policies formalize a 
change management strategy across 
the enterprise. Policies are fully 
enforced and consistently promoted. 


The organization actively follows a 
standards-based management plan 
after receiving certifications and 
accreditations. 


ADL INITIATIVE 


Key Process 


Area 
GOVERNANCE 


How mature 

is governance 

in support of 
innovative learning 
technologies? 


EXTENSIBILITY 


How mature is the 
IT environment 
for deploying 
innovative learning 
technologies? 


UBIQUITOUS 
LEARNING 
ENVIRONMENT(S) 


Does the 
organization have 
policies to afford 
education and 
training across 
time, space, path, 
mode, and access? 
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Activities deploy their own 
IT infrastructure without 
common planning. There 
are disparate networks and 
learning technologies. 


IT Infrastructure supports 
open and closed solutions but 
does not adhere to specific 
guidance or standards. 


Learning environments are 
strictly facilitated through 
organizational assets. There 
are no clear ubiquitous 
learning policies and defined 
objectives to guide ubiquitous 
learning. 


Technological Infrastructure 
The organization's technological-enabled Learning architecture: instructional systems, 
interoperability standards, software services 
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Organizational components 
maintain a common IT 
infrastructure resulting in 
multiple network enclaves that 
don’t interact. Governance is 
managed at the local level. 


IT infrastructure supports 
open and closed solutions 
based on prescribed standards 
and guidance. 


Clear objectives are set 

to guide the development 

of a ubiquitous learning 
environment. There is a 
need to evaluate existing 
systems and implement pilot 
prototypes. 


N= 


An enterprise infrastructure 
facilitates distributed 
education and training 
learning activities. Standard 
and configurable IT 
infrastructures are used 

to host dedicated learning 
capabilities. Policies 
manage the acquisition 

and maintenance of key IT 
systems. 


A pervasive IT infrastructure 
supports standards-based 
architecture solutions. 
Organizational policies drive 
IT requirements and define 
the process for acquiring, 
integrating, and accrediting 
new IT infrastructure. 


Investment in ubiquitous 
learning technologies grows 
along with the development 
of clear guidelines aligned to 
the organization’s core and 
technical visions. 
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An enterprise infrastructure 
facilitates distributed 
education and training delivery 
and human performance 
improvement. Governance 
results in globally managed and 
compatible learning systems 
and records accessible on a 
common network. Policies 
manage the acquisition and 
maintenance of key IT systems 
and technology insertions. 


Extensible services support 

any learning modality (e.g., 
simulator platforms, cyber 
range) and the specialized IT 
requirements to support each 
activity. Organizational policies 
drive IT requirements and 
define the process for acquiring, 
integrating, and accrediting 
new IT infrastructure. 


Best practices have been 
defined and implemented. 
Methods of measuring 
effectiveness of ubiquitous 
learning systems are developed 
and put in place. Procedures 
and policies are refined and 
improved to reflect changes in 
technologies. 
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Level 4 


An enterprise infrastructure facilitates 
distributed education and training 
delivery, in conjunction with external 
partners. Organizational governance 
results in globally managed, standards- 
based IT infrastructure that supports 
emerging learning technologies. 
Organizational policies manage the 
acquisition and maintenance of key IT 
systems and technology insertions. 


Extensible services to support any 
learning modality (e.g., simulator 
platforms, cyber range) and the 
specialized IT requirements to support 
each activity. Organizational policies 
drive IT requirements and define the 
process for acquiring, integrating, 

and accrediting new IT infrastructure. 
Federated data management and cloud 
hosting is driven by policy throughout 
the organization. 


The organization is constantly 
evaluating its ubiquitous learning 
environment to ensure continuous 
improvement and optimization. 


ADL INITIATIVE 


Key Process 


Area 
PRIVACY 


What is the 
maturity of policies 
and systems to 
protect private 
information? 


LEARNER IDENTITY 
MANAGEMENT 


How pervasive 
is the learning 
identity 
management 
solution? 


Key Process 


Area 
DATA STRATEGY 


Is there an 
education and 
training strategy 
for managing data? 


Privacy management is locally 
defined and follows industry 
standards. 


Local sign-on credentials are 
required per application or 
computer/device. 


Level 0 


Data from education and 
training activities are captured 
using disparate tools and 
technologies with proprietary 
data formats. 


Technological Infrastructure (continued) 
The organization's technological-enabled learning architecture: instructional systems, 
interoperability standards, software services 


Policies and procedures are 
in place to detect privacy 
breaches reactively, support 
remediation, and audit the 
information disclosed. 


Local sign-on credentials are 
required per organization. 


Privacy management is 
organizationally defined and 
enforced at all levels. 


A secure token-SSO, like a 
Common Access Card (CAC), 
is used only within the closed 
government infrastructure. 


Data Infrastructure 
The organization's infrastructure to promote data sharing and usage 


Some education and training 
activities capture learner 
performance data in xAPI 
format but are not integrated 
into a common LRS. 


xAPI is required by policy 

for all education and training 
activities to capture learner 
performance data. LRMI 
formatted metadata is 
required by policy to describe 
learning resources (activities, 
content, experiences, and 
courses). 


Level 3 


Reactive and proactive policies 
and controls are in place to 
prevent, detect, and respond 
to loss of privacy information. 
Effectiveness of controls are 
quantifiable. 


A secure token-SSO, like 
ID.ME, is used within the 
closed government and private 
infrastructure. 


Level 3 


xAPI is required by policy for all 
education and training activities 
to capture learner performance 
data. LRMI formatted metadata 
is required to describe learning 
resources (activities, content, 
experiences, and courses). 
Competency Frameworks are 
required to use the Reusable 
Competency Definition 

(IEEE 1484.20.1) to define 
competencies, the relationship 
between competencies within 

a competency framework, 

and the alignment of different 
types of evidence to assess 
proficiencies. 
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Level 4 


Privacy management establishes a 
clear understanding and a positive 
relationship with learners and their 
data. Formal processes are in place for 
regular auditing. The policy enforces 
user control of data, including the right 
to be forgotten. 


Federated SSO is required across 
disintermediated enclaves. 


Level 4 


xAPI is required by policy for all 
education and training activities to 
capture learner performance data. 
LRMI formatted metadata is required 
by policy to describe learning resources 
(activities, content, experiences, and 
courses). Competency Frameworks 
are required by policy to utilize the 
Reusable Competency Definition (IEEE 
1484.20.1) to define competencies, 
the relationship between competencies 
within a competency framework, and 
the alignment of different types of 
evidence to assess proficiency for a 
particular competency. An Enterprise 
Learner Record is required to support 
credential management, global learner 
attributes, and local learner profiles 
that contain the chain of evidence for 
earned credentials. 


ADL INITIATIVE 


Key Process 


Area 


DATA 
MANAGEMENT 
How well is 


data managed 
throughout the 
organization? 


DATA-DRIVEN 
DECISIONS 


How formalized 

is the review of 
data when making 
education and 
training decisions? 


DATA 
INTEROPERABILITY 


How consistent 
are the policies 
governing data 
interoperability? 


HUMAN 
RESOURCES 


How integrated are 
the education and 
training systems 
with the HR 
systems? 
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Management of data is a 
proprietary component of 
disparate software systems 
across the organization 

as defined by commercial 
technology implementation 
plans. 


Decision-makers do not 
consider data analytics 
in education and training 
planning. 


Organizational components 
procure education and training 
systems as needed without 
overarching policy dictating 
how education and training 
data will be promulgated 
through the organization. 


There is no Credential 
Management within the 
organization. Credentials 
are managed ad-hoc or by 
individuals. 


Data Infrastructure (continued) 
The organization's infrastructure to promote data sharing and usage 


Level 1 


Some subordinate units have 
mature data management 
within their systems, but 
this has not been normalized 
across the organization. 


Data analytics are considered 
on an ad-hoc basis across the 
organization. 


Organizational education and 
training systems use common 
commercial and government 
developed data standards. 
Interoperability is primarily 
focused on the exchange of 
data between systems for 
internal consumption. 


Minimal credential 
management capabilities 

are available within the HR 
system. Credentials are listed 
but not aligned with learning 
opportunities or career path 
planning. 


Level 2 


There is ad-hoc use of 
federated data internal or 
external to the organization. 
Formal processes are 
established for obtaining 
Authorities to Connect to 
disparate systems. 


Data analytics are used by 
high-level decision makers on 
an ad-hoc basis. 


Organizational education and 
training systems are required 
by policy to use commercially 
developed data standards and 
store these data on a common 
network so they are available 
for other education and 
training tools. 


A credential management 
capability is required by policy 
to support CTDL to enable a 
common way of describing the 
competencies represented by 
each earned credential. 


Level 3 


Policy guides federated data 
systems. Formal processes 
are established to enable 
access, authentication, 

and anonymization while 
also ensuring protection 

of Personally Identifiable 
Information (PII) and 
cybersecurity. 


There is systemic use of 

data analytics across the 
organization by high- and mid- 
level decision makers. 


Interconnected education 

and training systems use 
commercially developed data 
standards and store data on a 
common network available for 
other education and training 
tools. All data transforms are 
saved to preserve semantic 
meaning from the perspective 
of the source from their 
interoperable data. 


An established credential 
management capability 

is required by policy to 

support CTDL. Competency 
Management capabilities collect 
evidence from education and 
training systems and are 
available for use within the HR 
system. 
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Level 4 


Policies drive federated data 
strategies with an enterprise 
approach to federated identity and 
access management for evidentiary 
chains of learner performance data. 
Formal processes enable access, 
authentication, and anonymization 
while also ensuring protection of PII 
and cybersecurity. Enterprise services 
enable data owners to maintain 
ownership while sharing across the 
organization. 


Data driven decision making is 
institutionalized across the organization 
and is managed by policy. Analytics 
and tools are accessible by anyone in 
the organization. 


Internal and externally connected 
education and training systems are 
required by policy to use commercially 
developed data standards and store 
these data within a data lake that 
supports federated access. All data 
transforms are saved to preserve 
semantic meaning from the perspective 
of the source from their interoperable 
data. 


Centralized credential management 
capability generates badges based 

on individual performance in the 
operational environment (e.g., 
performance reviews, integration with 
digital systems). Enterprise adoption of 
competency-based learning principles. 


ADL INITIATIVE 


APPENDIX B: REVIEWED CMM MODELS 


OF \itelo lay 


BUSINESS 
INTELLIGENCE 


EDUCATION & 
TRAINING 


INTEROPERABILITY 


Enterprise Business 
Intelligence Maturity Model 
(EBI2M) 


Gartner Enterprise 
Information Management 
(EIM) Maturity Model 


Ladder of Business 
Intelligence (LOBI) 


TDWI's Analytics Maturity 
Model 


The Data Warehouse 
Institute’s (TDWI’s) Business 
Intelligence Maturity Model 


Defence Technology Enabled 
Learning (TEL) Maturity 
Model 


E-Learning Maturity Model 
(eMM) 


eQETIC Maturity Model for 
Online Education 


M-Learning Maturity Model 


Information Systems 
Interoperability maturity 
Model (ISIMM) 


Levels of Conceptual 
Interoperability Model (LCIM) 


Maturity Model for Enterprise 
Interoperability 


Organisational 
Interoperability Model 


Citation 


Chuah, 2010; Chuah & 
Wong, 2012 


Newman & Logan, 2008 


Cates et al., 2005 


Halper & Stodder, 2014 


Eckerson, 2009 


Defence Academy of the 
United Kingdom, 2019 


Marshall, 2010; 2013 


Rossi et al., 2015 


Alrasheedi & Capretz, 
2013 


Van Staden & Mbale, 
2012 


Tolk & Muguira, 2003 


Guedria et al., 2015 


Clark & Jones, 1999 


Levels 


initial, managed, defined, 
quantitatively managed, 
optimizing 


unaware, aware, reactive, 
proactive, managed, effective 


facts, data, information, 
knowledge, understanding, 
enabled intuition 


nascent, pre-adoption, early 
adoption, corporate adoption, 
mature/visionary, with a chasm 
between 3 and 4 


infant, child, teenager, adult, 
sage (sometimes level 0 of 
prenatal) 


ad hoc/initial, repeatable, 
defined, managed and 
measured 


not practiced/not adequate, 
partially adequate, largely 
adequate, fully adequate 


sufficient, intermediate, global 


preliminary, established, 
defined, structured, continuous 
improvement; descriptions for 
each level 


manual, ad-hoc, collaborative, 
integrated, unified 


none, technical, syntactic, 
semantic, pragmatic, dynamic, 
and conceptual 


5 levels: 0-4: unprepared, 
modeled, adhered, mapped, 
accommodated 


Independent, ad-hoc, 
collaborative, integrated, unified 


Dimensions/Categories 


Key processes are described per 
level, not grouped into dimensions/ 
categories 


vision, strategy, metrics, 
governance, organization roles, life 
cycle, accessible infrastructure 


3 process areas: technology, 
process and people 


Organization, Infrastructure, Data 
management, analytics, governance 


Scope, sponsorship, funding, value, 
architecture, data, development, 
delivery 


Governance, Strategy, Leadership, 
User, Integration: Content; 
Integration: Technology 


Delivery, planning, definition, 
management, optimization 


Didactic-Pedagogical; Technology; 
Management; Support; tutorial; 
Evaluation 


Key processes are described per 
level, not grouped into dimensions/ 
categories 


data interoperability, software 
interoperability, communication 
interoperability, physical 
interoperability 

The levels are divided into three 
categories: integrability (LO- 

2), interoperability (L3-4), and 
composability (15-6) 
"Categories: Conceptual, 
Technological, Organizational; 
KPAs for each dimension: Business, 
Process, Service, Data 


Preparedness, Understanding, 
Command Style, Ethos 


Included in 
Final Review 


No 


No 


No 


Yes 


No 


Yes 


Yes 


No 


No 


No 


No 


No 


No 
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Reason 


Still in development. One 
qualitative pilot study 


Information not readily accessible 


Incomplete, not well documented, 
criteria for evaluating maturity 
levels are not well defined. Highly 
IT-specific, other BI components 
are excluded 


Meets documentation and weak 
test requirements 


Older version updated by the TDWI 
Analytics MM 


Although still in development, 
included due to its relevance to the 
DL-CMM 


Developed model, provides 
documentation, meets strong test 


Preliminary model, not well 
documented; criteria for evaluating 
maturity levels not well-defined 


Incomplete, not well documented, 
criteria for evaluating maturity 
levels are not well defined. 


Still in development; pilot study 


Technical reference model 


Still in development. Future work is 
in planning to perform case studies 
in enterprises 


Technical reference model 


ADL INITIATIVE 


Reviewed CMM Models (continued) 


Citation 


Category 


HUMAN People's Capability Maturity ‘ 
RESOURCES ~— Model CURBS St aeeee 
Business continuity 
management (BCM) maturity 
model for the UAE banking Reales i 
sector 
Corporate Performance 
Management (CPM) Capability Aho, 2009, 2012 
Maturity Model 
ORGANIZATION 
InfoSys KM Maturity Model Kochikar, 2000 
Enterprise Architecture Government Accounting 
Management Maturity Office, 2010 
Framework (EAMMF) i 
Capability Maturity Model Paulk et al., 1991, Paulk 
(CMM), v.1, 1.1, 2C et al., 1993, Paulk 1997 
SOFTWARE 
DEVELOPMENT/ Capability Maturity Model CMMI Product Team. 
TECHNOLOGY Integration (CMMI), v.1.1, (2002). 


IT Performance Measurement 


Maturity Model ee es 


Levels 


initial (inconsistent 
management), managed 
(people management), defined 
(competency management), 
predictable (capability 
management), optimizing 
(change management) 

ad hoc, managed, defined, 
integrated, optimized; 2nd 
access: BCM process quality; 
BCM process scope 


unaware, ad-hoc, repeatable, 
defined, managed, optimized 


default, reactive, aware, 
convinced, sharing 

Each level is characterized 

by three factors: knowledge 
acquisition, knowledge 
dissemination, knowledge reuse 


7 Stages 


initial, repeatable, defined, 
managed, optimizing 


initial, repeatable, defined, 
managed, optimizing 


non-existent, initial, repeatable, 
defined, managed, optimized 


Yi nat=yarssrelasyA@r-ik-1e [eat =1s 


Each maturity level is composed of 
several process areas. Each process 
area contains a set of interrelated 
practices that satisfy a set of goals 
for achieving the maturity level. 


BCM program management; 
planning and analysis, development 
of the BCP, implementation, 
maintenance 


Management & Organization, 
Technology, People & Culture, 
Processes 


People, Process, Technology 


4 representations, core success 
attributes for each representation 


Key processes are grouped by level 


Process Management, Project 
Management, Engineering, Support 


Still in development 


Included in 
Final Review 


Yes 


No 


No 


No 


Yes 


Yes 


Yes 


No 
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Reason 


Built on CMM, CMMI. Well 
documented, published research 
on use cases 


Incomplete, not well 
documented, criteria for 
evaluating maturity levels are 
not well defined 


Provides a quick way for 
organizations to see where they 
are and where they need to go 
next. Works as a communication 
and change management tool 


Not well documented; criteria for 
evaluating maturity levels are 
not well defined 


Published results, relevant to 
DL-CMM 


Most maturity models are based 
on this CMM and subsequent 
models from the Capability 
Maturity Model (CMMI) Product 
Team 


Developed model, provides 
documentation, meets strong 
test 


Incomplete, not well 
documented, criteria for 
evaluating maturity levels are 
not well defined 


/hvanced|Distriouted|learning 


